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We study various Mott insulating phases of interacting spin-- fermionic ultracold atoms in two- 
dimensional square optical lattices at half filling. Using a generalized one-band Hubbard model with 
hidden SO(5) symmetry, we identify two distinct symmetry breaking phases: the degenerate anti- 
ferromagnetic spin-dipole/spin-octupole ordering and spin-quadrupole ordering, depending on the 
sign of the spin-dependent interaction. These two competing orders exhibit very different symme- 
try properties, low energy excitations and topological characterizations. Near the SU(4) symmetric 
point, a quantum critical state with a 7r-flux phase may emerge due to strong quantum fluctuations, 
leading to spin algebraic correlations and gapless excitations. 

PACS numbers: 71.10.Fd, 02.70.Ss 



I. INTRODUCTION 

The Mott insulating states are attracting extensive at- 
tention in modern condensed matter physics, which are 
believed to play a crucial role for resolving the myster- 
ies in high temperature superconductors, heavy fermion 
compounds, colossal magnetoresistance manganites and 
so on. In the conventional band theory, the insulators 
are characterized by an energy gap between the highest 
filled and the lowest empty states. On the other hand, a 
charge excitation gap can open up with a partially filled 
energy band due to the competition between kinetic en- 
ergy and repulsive force between the particles. Since the 
strong correlation effects are difficult to handled, an effi- 
cient theoretical description of the Mott insulating states 
is still a challenging task. 

Since Greiner et al. experimentally succeeded in ob- 
serving the superfluid-Mott phase transition of ultracold 
bosonic atoms in optical lattices [![, studying strongly 
interacting cold atom systems becomes a new research 
direction of condensed matter physics. In fact, an atomic 
Mott insulator is prepared using a deep optical lattice, 
where the particle number fluctuations are strongly sup- 
pressed. Such systems are rather clean and controllable 
compared to conventional solid state systems. Besides 
the single component atoms, the spinor atoms can also 
be loaded into the optical traps or lattices, leading to a 
variety of novel quantum phenomena. 

Depending on the hyperfine spin values, the ultra- 
cold atoms are classified as bosons with integer F and 
fcrmions with half odd F. Most previous studies on 
the Mott insulating states focused on scalar and spinor 
bosonic atoms. Various Mott insulating phases are also 
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formed in higher spin ultracold atoms in optical lattices. 
For spin-1 bosons, the spin-singlet, dimerized, and the ne- 
matic Mott insulating states were studied @, H, S @1 • 
For spin-2 bosons, even more complicated Mott insula- 
tors can be realized @, @|. Depending on the scatter- 
ing length and particle occupation number per site, the 
ground state can be spin-ordered ferromagnetic, cyclic 
and nematic states. When the potential barriers of the 
optical lattices are very high, one can even observe the 
coherent spin-mixing oscillations of the two isolated spin- 
2 87 Rb atoms in one site 0, Recently, the Mott 

insulating phases of spin-3 52 Cr atoms have attracted at- 
tention motivated by the experimental progress, where 
the long-range dipole interaction plays an important role 
in determining the exotic ground states [Til ]. 

Actually, the physics of fermionic ultracold atoms in 
optical lattices is also interesting. For spin-i fermions, 
the Hubbard and t — J like models can be realized in 
optical lattices, which may shed light on the fundamen- 
tal issues in studying the high-T c superconductors in 
the doped cuprates [12J. Furthermore, the high spin 
fermionic atoms provide opportunities to study novel ba- 
sic physics with no counterparts for electrons in solids. 
Recently, rare-earth 173 Yb atoms with the lowest hyper- 
fine manifold F — I have been cooled down to quantum 
degeneracy regime [l3j |. free of spin relaxation. On the 
other hand, the F = I interacting fermionic system is at- 
tracting theoretical interest, which can be realized with 
alkali atoms 132 Cs, as well as alkaline-earth atoms 9 Be, 
135 Ba, and 137 Ba. Recently, Wu et al. noticed that the 
spin- 1 fermions with s-wave scattering interaction enjoy 
a hidden SO(5) symmetry, without tuning any parameter 
Then, more interesting issues are studied, such 
as the competition between the baryon-like quartet pair- 
ing state and quintet pairing state in one-dimension [ll| , 
the topological generation of quantum entanglement by 
non-Abelian half quantum vortices in quintet superfluid 
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phase [l7| . 

In this paper, we will investigate the various Mott- 
insulating states and map out the ground state phase 
diagram for a half filled spin-| ultracold fermionic atom 
system on a two dimensional square optical lattice. In 
addition to the spin-quadrupole ordering state [HI, we 
will show that a new Mott insulating state with degen- 
erate antiferromagnetic spin-dipole/spin-octupole order- 
ing can be formed as well. Between these two distinct 
spin-ordered phases, an SU(4) 7r-flux state may emerge 
as a quantum critical phase due to the competing orders. 
Furthermore, the spin-quadrupole correlations, as well 
as the spin-dipole and spin-octupole correlations display 
the same power-law behavior. This is in analogy with the 
staggered flux state in quantum spin S = | Heisenberg 
antiferromagnets, in which the competing Neel order pa- 
rameter and the valence-bond solid order parameter ex- 
hibit the same power-law correlations. In fact, the SU(4) 
7r-flux state and the SU(2) staggered flux state both be- 
long to a class of algebraic spin liquid states that unify 
the competing orders by emergent symmetry at low en- 
ergies [19]. A variety of novel algebraic spin liquid states 
have been studied in the underdoped region of cuprates 
[20| , toy spin models [U [22|, [23|, [24]] and strongly corre- 
lated p-orbital band bosons in optical lattices [2a ]. 

The paper is organized as follows. In Sec. II, the 
general spin-| Hubbard model will be introduced, which 
describes the spin-| interacting fermionic atoms in opti- 
cal lattices. In Sec. Ill, we will briefly review the Mott 
insulating phase with a staggered spin-quadrupole order. 
In Sec. IV, the Mott insulating phase with degenerate 
antiferromagnetic spin-dipole/spin-octupole order will be 
studied within the functional integral approach. The cor- 
responding symmetry breaking pattern and the low en- 
ergy collective excitation modes are discussed. Sec. V 
is devoted to an analysis of the SU(4) 7r-flux state as a 
quantum critical state. The phase diagram is obtained 
based on an effective model at low energies. Finally, a 
brief discussion is presented in Sec. VI. 



where ip is a four-component spinor defined by ipi = 

(^,3/2,^,1/2,^,-1/2,^,-3/2) and the chemical poten- 
tial \x — is fixed at half filling on a bipartite lattice. 
Within the harmonic approximation, the parameters for 
the two-dimensional square optical lattice are given by 



=£^ 3 exp(-2£ 2 ), 



C2 



5a 2 — a ^ 3 



A 
A 



E R e, 



(2) 



where Er = hk 2 /2m is the recoil energy, £ = (Vx/Er) 1 / 4 
and as is the s-wave scattering length in the total spin- 
s' channel. Due to the Pauli's exclusion principle, the 
s-wave interactions of identical spin-| fermions in total 
spin S — 1, 3 channels are forbidden. The particle num- 
ber and spin operators at lattice site i are defined by 

Ni=^2 V'iaV'ia, S 4 = ^L S ^i0, (3) 
a a/3 

where S are the spin-| matrices. In addition, we have to 
consider the higher-order products of the spin matrices. 
The spin-quadrupole matrices are defined by 

4= (s x s v + s y s x ) = r 1 , 4= (s z s x + s x s z ) = r 2 , 
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(s z s v + s y s z ) = r 3 , (s^) 2 -^ = r 4 , 



[(s x f - (syf] = r 5 



(4) 



where 



-il 
il 



2,3,4 



a 
-a 



,r 5 = 



/ 
/ 



II. GENERALIZED ONE-BAND HUBBARD 
MODEL 

An optical lattice is a periodic potential V(r) — 
J2 r=x y z V r sin 2 (k • r) created by standing- wave laser 
beams, where |k| = 27r/A is the wave vector of the laser 
and A/2 is the lattice spacing. When the potential is 
tuned to V z ^> V x = V y , the atoms are confined to the 
lowest Bloch band of a two-dimensional x-y plane square 
lattice at low temperatures. The system of spin-| ultra- 
cold interacting fermionic atoms is well described by a 
generalized one-band Hubbard model 



H 



(ij) 



i 



(i) 



are the five Dirac Gamma matrices satisfying the Clif- 
ford algebra {r a ,r 6 } = 2<5 a b, / is a 2 x 2 unit matrix 
and a are Pauli matrices. In terms of T matrices, we 
have also tensor operators defined by r afc = — |[r a ,r 6 ]. 
Thus, the spin-| (dipole) matrices are expressed as linear 
combinations of r ab : 



VJ? T 
2 



s x = ^r 34 + -(r 12 + r 35 ), 



S v = -^r 24 - i(r 13 -r 25 ), 

gz = _pl5 + i r 23 ; 



(5) 



which form the spin SU(2) algebra with [S 01 ^ 13 ] = 
ieap^S 1 . Moreover, there are seven spin-octupole ma- 
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trices given by 





(S x ) 3 




(S y ) 3 




(S z ) 3 


{S x ,(S y ) 2 


- (s z ) 2 } 


{S y ,(S z ) 2 


- (S x ) 2 } 


{S Z ,(S X ) 2 


- (s y ) 2 } 


S x S y S z 4 


- s z s v s x 



7y/3 



p34 , _p35 

8 



13^13 



7V3 T 



.24 , 1 -p25 



13, 



.13 



I£r 23 — -T 15 
8 4 ' 



.34 , _p35 

+ 2 ' 



Vj^-p24 _ 3p25 

2 2 ' 



^^45 



(6) 



where we have used the abbreviation {A, B} = AB + BA. 
The spin-quadrupole operators, spin-dipole and spin- 
octupole operators can be simply expressed as 



(7) 



< = (l<a<5), 

a/3 

L ? = ~\ E ^r Q >^ (1 < a < 6 < 5), (8) 



a/3 

where n™ corresponds to an SO (5) superspin vector, and 
Lf b constitute the adjoint representation of the SO (5) Lie 
group satisfying the commutation relations: 

[L° b , L cd \ = iSij (5 a dL b t c + 8b c L c l d - b ac L\ d - 5bdL°; c ) . (9) 

Furthermore, nf and L® b together as generators form an 
SU(4) Lie group, the highest symmetry group for four- 
component spin-| fermions in the particle-hole channel. 
Under the SO (5) transformations, the sixteen bilinear op- 
erators in the particle-hole channel are classified as parti- 
cle number operator (scalar), spin-quadrupole operators 
(vector), spin-dipole and spin-octupole operators (ten- 
sor). 

We would like to emphasize that in the multipolar 
expansion of classical electrodynamics, the higher-order 
multipolar interactions are usually much weaker than the 
dipolar interactions. However, this is not the case in the 
present system, where spin dipolar and multipolar oper- 
ators are all expressed in terms of bilinear combinations 
of the fermionic operators. 

In this paper, we will focus on the Mott insulating 
phases with two atoms per site. In the limit of t = 0, each 
lattice site decouples from its nearest neighbor sites, then 
the single site ground state is a spin singlet (Si = 0) for 
C2 > and a spin quintet (Si = 2) for c 2 < 0. For c 2 = 0, 
the ground state is a six-fold degenerate state and the 
interactions gain an enhanced SU(4) symmetry. Since 
the scattering lengths for the spin-| fermionic ultracold 
atoms are not available yet, we consider the parameter 
regime of cq |c2 1 , t , where the spin-dependent inter- 
action are much weaker than the spin-independent one. 



Actually, this request is well satisfied for current alkali 
atom experiments. Under these considerations, the sta- 
bility of Mott insulator with double occupancy is also 
guaranteed and an interesting phase diagram with three 
different kinds of Mott insulating states will be obtained. 



III. SPIN-QUADRUPOLE ORDERING STATE 

In this section, we briefly review the Mott insu- 
lating phase with staggered spin-quadrupole ordering 
formed by the half-filled spin-| fermionic atoms in a two- 
dimensional optical lattice [Tj|. To reveal such an order, 
the generalized Hubbard model can be rewritten in an 
SO (5) invariant form [l4[ 



3c 2 



E 

i,l<a<5 



a\2 



(O 



-^^E(^- 2 ) 2 -mE^ do) 



Provided the commutation rules 
a, 



[L1\n^=i5 t3 (5 bc n a t -5 ac n b ) 



(11) 



are satisfied, the generic SO (5) symmetry of this model 
manifests explicitly, i.e., the Hamiltonian is invariant un- 
der the transformation 



ip -> t/)' = e- lLa " e * b 4>, 1 < a < b < 5, 



(12) 



where 9 a b are the Euler angles representing infinitesimal 
rotations between the ten independent planes in five di- 
mensions. At half filling, the chemical potential [i is zero 
and the average number of fermions per site is (Ni) = 2. 
As far as an insulating state is concerned, the particle 
number fluctuations can be neglected and we focus on 
the quantum spin fluctuations, which result in various 
spin ordering states. In particular, for ci > 0, site- 
singlets formed by two spin-| fermions with total spin 
S = are energetically favorable. In this case, the 
quantum spin-quadrupole fluctuations are an important 
feature of the generalized Hubbard model. As we have 
shown in the previous work [18j, the doubly occupied 
fermionic atoms in a square lattice can exhibit a stag- 
gered spin-quadrupole ordering with the order parame- 
ter (— l) l (n1) 7^ 0. Without loss of generality, the order 
parameter can be chosen as (— \) % {nf) ^ 0, where 

2n t = ^3^,3 +V>l_3-0i_§ - ip\ 1^,1 -^LiV'i.-i, 

(13) 

corresponding to the difference between the S z — ±3/2 
and S z = ±1/2 spin densities in the site singlet 
state. The typical configuration of this particular spin- 
quadrupole ordering state is shown in Fig. 1. This 
quantum analogue of liquid crystal state breaks the spin 
SU(2) rotational symmetry, but preserves the time rever- 
sal symmetry. 
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FIG. 1: (Color online) The picture of the staggered spin- 
quadrupole long-range ordered state with the order parameter 
(—l)'{nj) on a two-dimensional square lattice, where each site 
is occupied by a spin singlet state. The long blue arrow and 
short red arrow denote a fermion with S z = ±| and S z = ±|, 
respectively. 



Due to the fixed direction n\ of the SO(5) superspin 
vector, the symmetry of the spin-quadrupole ordering 
ground state is spontaneously broken down to SO (4) and 
the Goldstone manifold is SO(5)/SO(4)=S 4 , which is a 
four-dimensional sphere. Since Lf b performs a rotation 
between nf and n\, the generators of the SO (4) symme- 
try are Lj 2 , Lf, Lj 5 , Lf , Lf, and Lf. Thus, there 
are four Goldstone bosonic modes without coupling to 
each other due to the residual SO (4) symmetry. The 
presence of this high symmetry qualitatively affects the 
strong coupling behavior of the spin-quadrupole density 
waves, whose velocity is saturated at large C2 as vq ~ t. 
However, in the antiferromagnetic ordering state of the 
half-filled spin- ^ Hubbard model, the Goldstone manifold 
is SO(3)/SO(2)=S 2 . The residual SO(2) symmetry does 
not forbid transverse spin wave mixing, and the density 
wave velocity is suppressed as vsdw ~ t 2 /U in the strong 
coupling limit. 

Another interesting physics in the spin-quadrupole or- 
dering state is the non-abelian Berry phase. Since the 
four-component SO (5) spinor forms a seven-dimensional 
sphere S 7 , the adiabatic rotation of the fermionic quasi- 
particle in spin-quadrupole ordering state defines the sec- 
ond Hopf map 



S 7 -> S\ 



(14) 



which enables an interesting non-abelian SU(2) gauge 
connection of the so-called Wilczek-Zee holonomy [2g, 
[27| . Mathematically, it is understood that S 7 is an SU(2) 
bundle over S 4 sphere. Therefore, the topological struc- 
ture of the spin-quadrupole ordering state is described by 
the second Hopf map and the second Chern number. As 
the fermionic quasiparticle spectrum is two-fold degener- 
ate, once we perform an adiabatic circuit rotation of the 
superspin vector nf, the spinor state can be a linear com- 
bination of the two degenerate low energy states, which 
is related to the original state by a 2 x 2 unitary trans- 
formation matrix. Actually, the SU(2) non-abelian Berry 
phase is also present in the quintet pairing state [28j and 



plays a key role in the entanglement generation between 
the fermionic quasiparticles and the half-quantum vortex 
0. 



IV. DEGENERATE ANTIFERROMAGNETIC 
SPIN-DIPOLE / SPIN-OCTUPOLE ORDERING 
STATE 

A. Saddle point approximation 

Next we will consider the case for C2 < 0, where a spin 
quintet state is formed by two spin-| fermions on each 
lattice site with total spin S — 2. Applying the SU(4) 
Fierz identity 

E (^f) 2 + E (nf) 2 + |(JVi-2) 2 =5, (15) 



l<a<fc<5 



Ka<5 



to the interactions of the model Hamiltonian, we can 
rewrite the generalized one-band Hubbard model as 



H = -t E WLVV+H.c.) 

+|E(^- 2 ) 2 -^E^ 



3C2 

4 



E w 



b\2 



,l<a<h<5 



(16) 



where the chemical potential /i is set zero at half filling. 
As far as the Mott insulating phase is concerned, the par- 
ticle number fluctuations can be neglected safely, and the 
spin-dipole and spin-octupole fluctuations become dom- 
inant when C2 < 0. 

Formally, the partition function is written as a path 
integral 



Z = 



J Vtp^Vip exp - J L (t) dr 



(17) 



where (3 = and the Lagrangian is given by L — 

Hi, a i ) t a ®T' t l > ia+H. In the following we denote U = -|c 2 
for simplicity and a Hubbard-Stratonovich transforma- 
tion is performed 



exp 



U 



T>(p exp 



f dr E(if) 

J " l ,a<b 

Jo 



b\2 



-M) 



where a ten-component order parameter field tp is intro- 
duced. By integrating out the fermion fields, an effective 
action is obtained as 



Scff = f dr E \{vt f - Tr In [8 T + M] 

^° i,a<b 



(18) 



where the trace is taken over the spinor space, the spatial 
and imaginary time coordinates. The matrix element of 
M and Green's function (GF) of the fermions satisfy 

(r i ,T,a!|M|r,-,T / ,/3) = -2t6 rr 'Si t j + sS a fi 

^A A 



a<b 



, abj^ab 



G Q /3(r 4 ,r;rj,r') = (r;,T,a| 



■1 



d T + M 



So far no approximations have been made. Once we per- 
form a loop expansion on the effective action with respect 
to if, the instability owing the negative coefficient of the 
quadratic term in (p is dominated by the so-called Fermi 
surface instability, which is similar to the well-known an- 
tiferromagnetic instability in half filled Hubbard model 

M- 

In order to determine the long range order of Mott 
insulator, we first consider the saddle point solution of 
the effective action. Differentiating S e s with respect to 
ipf b (r) , we obtain the equation 



a/3 



/3a- 



(20) 



Then it is expected that the ground state is in a staggered 
phase of the SO(5) adjoint order parameter, namely, a 
staggered degenerate spin-dipole /spin-octupole order- 
ing phase with a static order parameter field (pf b (r) — > 
\(p\e z< ^' Ti d ab , where Q = {%, tt) corresponds to the nest- 
ing vector in a two-dimensional square lattice, and d ab 
is the symmetry breaking direction. The degeneracy of 
spin-dipole and spin-octupole order is protected by the 
exact SO (5) symmetry of the Hamiltonian, which implies 
the spin-dipole operators can be rotated to spin-octupole 
operators and vice versa. Thus, this degeneracy is robust 
against quantum and thermal fluctuations. However, an 
anisotropy, such as lattice potential, will pin down a spe- 
cial choice of the ground state. Actually, the effective 
action at the saddle point becomes 



S (o) 



a<b 



</> k (^„K b r a V k _ Q (^„) \ , (21) 



fermion dispersion relation is £k 



cosky), and oj n is the fermionic Matsub- 



where the 
— 2i(cos k x + 

ara frequency. The fermion GF has to be defined as 
G a p (k, k'; r) = - (T T ^ ka (r)V>_ k / ^(0)) i which has non- 
zero off-diagonal terms in momentum space due to the 
umklapp processes with respect to Q. At the saddle 
point, the fermion single particle GF is given by 

G a( 3 (k,-k';iw„) 

(iu n + e k ) A Q/3 A kk , + Ju\<p\ J2 a<b d ab T a a b 6 k , M Q 
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FIG. 2: (Color online) The picture of the long range ordered 
state with the order parameter (-l) l (L\ 5 ) (a) and (-1)*(L? 3 ) 
(b) on a two-dimensional square lattice, where each site is 
occupied by a spin quintet state. 



where GF poles determine the fermionic quasiparticle 

12 At half filling, the upper 



spectrum = ± y e 
band is empty while the lower band is completely filled. 
Thus an energy gap A = 2w^|y>| opens up in the single 



particle spectrum. At T 
by 



OK, the gap equation is given 



11^1 
U ~ A 7 V^k' 

k 



(23) 



In the limit of U < t, it gives rise to A ~ 2te~ 7r v /2 */ c/ j 
which implies the Fermi surface is fully gapped for arbi- 
trary small but finite U. For U ^S> t, we have A ~ 2U, 
i.e., a large Mott gap. 

We have shown two typical configurations of this long- 
range ordering phase in Fig. 2, and the corresponding 
order parameters are denoted by (— l) l (L} 5 ) and 
(— l) l (L 23 ) ^ 0, respectively. More explicitly, L} 5 and 
L 23 are expressed as 



2Lf = ^sV'a+^i^i 

-2 2 '2 2 

2Lf = vliV'a+^Ls^,. 



I, 2 



I, 2 



corresponding to two different patterns of total S z — 
±2 and S z = ±1 of two atoms in the site-quintet state, 
respectively. Actually, both configurations display the 
coexistence of antiferromagnet spin-dipole and staggered 
spin-octupole ordering with breaking the spin SU(2) and 
time reversal symmetries. 

To simplify the following discussions, we assume the 
order parameter is fixed by (— l) l (L,p) ^ 0. Along 
with the order parameter Lj 5 acquires non-zero vac- 
uum expectation value, the SO (5) generators Lj 5 , 
L 23 , L 24 and £ 34 still keep the ordering state invari- 
ant, where the latter three SO(5) generators form an 
SU(2) subgroup. Therefore, the Goldstone manifold is 
SO(5)/[SU(2)xU(l)]=CP 3 , which is a six-dimensional 
complex projective space. According to the Goldstone 
theorem, there will be six branches of Goldstone bosons 
induced by the spontaneous symmetry breaking. 
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The topological aspects on the degenerate spin- 
dipole/spin-octupole ordering phase are also interesting. 
Mathematically, a seven-dimensional sphere S 7 can be 
viewed as a U(l) bundle over CP 3 , which is the Gold- 
stone manifold in the current case. Because the four- 
component SO(5) spinor lives on S 7 , the adiabatic circuit 
rotation of the fermionic quasiparticles in the degenerate 
spin-dipole with spin-octupole ordering state defines a 
U(l) gauge connection. Although the fermionic quasi- 
particle spectrum is also two-fold degenerate, once an 
adiabatic circuit rotation of the order parameter Lf b is 
performed, only an additional U(l) phase factor can be 
generated in the resulting spinor state, instead of lin- 
ear combination of the two degenerate states due to the 
abelian nature of the U(l) Berry phase [3(|. Therefore, 
unlike the spin-quadrupole ordering state, the topologi- 
cal structure of the degenerate spin-dipole /spin-octupole 
ordering state is characterized by the U(l) abelian Berry 
phase and the first Chern number. 

B. Gaussian fluctuations 

To further study the collective excitations, we consider 
Gaussian fluctuations around the saddle point solution, 



ip = ip c + Sip. Then the effective action S e g can be ex- 
panded as S e g = X^^Lo ^ (Vci Sip) from 

Trln [d r + M] = Trl^-G^ 1 ) - ^ — Tr(GV)", (24) 

n 

where G represents the fermion GF at the saddle point 
and the matrix element of V is given by 

(r,, r, a|V|r„ r' , /?) = S^^^Sipf ( r ) 

a<b 

(25) 

Since V only contains a linear term in dip, the above 
procedure is indeed an expansion in the spin-dipole and 
spin-octupole fluctuations. The first order term in Sip 
vanishes due to the saddle point equation. The Fourier 
transformation of 5ipf b (r) is defined by 

Sipf ( r ) = _j= £ 6 < (^) e 4p '™ T , (26) 

where u>i are bosonic Matsubara frequencies. After some 
straightforward algebra, up to the second order expan- 
sion in the fluctuation field Sip, we arrive at 



s(2) = {^o(p,^)* t p(-^)$p(^) 

+ ]T if 1 (p, iW 0[^ Q P (^0^p a (^0 + ^p(-^)^p 5 (^)] 

a=2,3 : 4 

+ J2 ^( P , m)[5ip%_ Q (-m) 5vf (i"i) ~ *V-p-q i~m) ^p" (<wj)]} 

a=2,3,4 

where $i = (Sip} 5 , Sip 23 , Sip 24 , Sip 34: ) T is a Nambu spinor and the kernel functions are given by 

K (p,iuji) = 



U ^ Lu n (uj n +ui) - e k £k+p 



4 



(3N 



k,to„ K + K) K + LJi) + E 2 +p 



2£i(p,«wi) 
K 2 (p,iuJi) 



1 

2 

UA 

2f3N 



U ^ Lu n (uj n +ui) - e k £k+p + — 



(3N 



k,**, K + K) K + wi) + E 2 +p 



y 



(27) 



(28) 

(29) 
(30) 



In the low energy limit iuoi — > 0, for the momentum transfer p = Q, K (Q, 0) > and K\ (Q, 0) = K 2 (Q, 0) = 
following from the gap equation. Therefore, the collective excitation modes in the first term of Eq. (f2"Tj) is gapped, 
and the remaining terms describe the Goldstone modes related to the symmetry breaking. From Eq. (l27j) . we notice 
that the couplings between the fluctuating Goldstone fields are in a twin form as Sip 12 Sip 25 , Sip 13 Sip 35 and 5ip 1A Sip A5 . 
As we discussed, the resulting effective action enjoys a residual SU(2)xU(l) symmetry and the coupling between the 
Goldstone fields are allowed. 

In order to evaluate the low energy behavior of the Goldstone bosons, we perform the summation over the Matsubara 
frequency in the kernel functions of K\ , K 2 and analytical continuation iui — > lo + i0 + . When the long wavelength 
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and low energy limit are considered at T = OK, the kernel functions are expanded to the leading order in q and uj as 

(31) 



tfi(Q + q,w) 
K 2 (Q + <l,u)) 



aq 2 — boj 2 , Ki(c[,lu) w d, 



where the coefficients are given by 

r d 2 k Ut 2 sin 2 k x 



2El 



(2^) 2 

Thus, the effective action describing the Goldstone bosons is given by 

/ " ' ' q Q ' ■ ; \ ' 

dqduj 



r d 2 k u _ r d 2 k mA r d 2 \ 

J_„ (2n) 2 8El' C -- J_„ (2nr 8^ ' J-« (2tt 



k Uel 



Sgb = 



E 



/^L- ' Q (-w)\ 

^l 5 q _ Q ("W) 

V ^ q (-w) y 



/ aq 2 - bar 



— 





d CbJ 

OjJ aq 2 — 





CUJ 





K+Q (W) 



(32) 



(33) 




FIG. 3: The degenerate spin-dipole/spin-octupole density 
wave velocity v (dotted line) is compared with the spin- 
quadrupole wave velocity vq (solid line) . 



Then the dispersion relation of the Goldstone bosons is 
determined by the pole of the collective mode correlation 
functions, giving rise to u = v\q\, a linear dispersion with 
the density wave velocity 



ad 



bd 



(34) 



The corresponding numerical results are displayed in Fig. 
3. 

In the small U limit, the density wave velocity is ap- 
proximated bynK 2i(2J7/t) 1 / 4 /y / 7r, while in the large U 
limit, it is given by v w 2\f2t 2 jU . Due to the presence 
of the coupled vibrations of the transverse modes, the 
wave velocity is suppressed in the strong coupling limit, 
in a sharp contrast to the velocity of the spin-quadrupole 
density waves. These properties can be understood on 
the basis of the symmetry considerations. The trans- 
verse mode couplings in spin-quadrupole density waves 
are absent because the SO (4) invariance of the effective 



action under the Gaussian fluctuations does not allow 
such couplings. However, the residual SU(2)xU(l) sym- 
metry does allow coupled vibrations of the Goldstone 
modes, which is analogous to the transverse spin wave 
mixing in half filled spin-i Hubbard model. 



V. ALGEBRAIC SPIN LIQUID STATE 

In the previous sections, we have established the spin- 
ordered Mott insulating phases of spin-| fermions in 
various limits. Near the special SU(4) point, the ener- 
gies of the ordering states are very close to each other. 
Since these two competing ordering ground states ex- 
hibit quite different symmetry properties, low energy ex- 
citations and topological characterizations, a continuous 
quantum phase transition from one to another is almost 
impossible. Therefore a nontrivial quantum critical state 
may emerge around the SU(4) symmetric point. 



A. SU(4) symmetric point 

Let us first consider the SU(4) point (c 2 = 0). In this 
case, the generalized one-band Hubbard model at half 
filling is reduced to the SU(4) Hubbard model: 



H = -t ]T (4 a ^ a + H.c.) + f J2( N * 2 ) 2 - ( 35 ) 

(ij),a i 



When the strong coupling limit (c 3> t) is considered, 
the effective exchange model can be derived by a second- 
order perturbation theory, which reads 

h cS = jJ2( E L ? L f+ E n °»°)' ( 36 ) 

(ij) l<a<6<5 l<a<5 

where the coupling constant J ~ t /cq. When we intro- 
duce a spinon representation in terms of four component 
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fermions, the operators nf and Lf b can be expressed as 

a/3 

L t = ~ \ E /Ws/tf (1 < o < 6 < 5), (37) 



a/3 



with a local constraint ^ Q f} a fi a = 2. The effective 
model Hamiltonian can thus be written as 



H cS - - J fiafjafjpfiff- 



(38) 



This is nothing but the SU(4) quantum antiferromagnetic 
Heisenberg model, which was studied in the large- ./V limit 
on a square lattice by Affleck and Marston [3l[ . Formally, 
we can express the partition function as a path integral 



Z = / VpVfVXexp 



L e g(T)dT 



(39) 



where the effective Lagrangian is given by 

L cS = J2 4a/*« +#c ff +£ \i nr, flfio - 2 ) , 

i,a i \ a / 

and the Lagrange multiplier field A./ is introduced to im- 
pose the local constraint. Then the interaction part can 
be decoupled via a valence bond auxiliary field 



exp 



[ dr £ flfjaf^fiff 

JO . n 



T>xef° ^ dr [^<.j),-(/, t Q /^Xj I +c.c.)-E( i3 -) lxi J l 2 ]^ 41 ) 



where X ij — Xji — \Xij\ e ~ laii an d a ij is a phase field. 
Under the gauge transformation — > + 9i — 8j, 
Si fi el6i , the effective action is invariant, because d T 9i 
term can be absorbed in the Lagrange multiplier. The 
gauge invariance thus gives rise to a U(l) gauge field, 
which is also related to the local constraint. By gauge 
fixing, Xij can be chosen to be real, and a saddle point 
approximation leads to the 7r-fiux state with 



Xi,i+x = -x(-l) S , Xi,i+y 



(42) 



and the Lagrangian multiplier is fixed by the particle- 
hole symmetry at A, = 0. Although the above mean-field 
ansatz breaks the lattice translation symmetry, it can be 
restored when projecting the mean-field state onto the 
physical Hilbert space with two spinons per site. Then 
the effective model can be written as 

H vF = 2JN X 2 - JxJ^ifLfHw + fLfi-v,* 

iot 

-(-l^/l/i-x.a ~ (-If* fth +x , a ), (43) 



which is quadratic in spinon operators and is easily diag- 
onalizcd. The spinon excitation spectrum is obtained by 
= ±2Jxa/cos 2 k x + cos 2 k y with four gapless nodes at 
k x = ±5 and k y = ±5. The ground state energy per site 
is determined by filling the negative spinon bands: 



e^p = 4 



dk x 
2^ 



(44) 



where the minimum energy is attained at X = 0.958091. 
When taking into account the nodal SU(2) symmetry in 
the low energy limit, the 7r-flux state gains an emergent 
SU(8) symmetry [Tg. 

In fact, the low energy effective theory of the SU(4) tt- 
flux states should be described by a fluctuating massless 
U(l) gauge field minimally coupled to the Dirac spinons. 
This is different from the 7r-flux state with a gapless 
SU(2) gauge field in the spin- 1/2 quantum Heisenberg 
antifcrromagnets. However, the SU(4) 7r-flux state re- 
sembles the SU(2) staggered flux state. Recently, Her- 
mele et al. argued that if the spin index is generalized 
to N, the problem of 2N two-component Dirac fermions 
coupled to a compact U(l) gauge field is deconfined for 
sufficient large N 32]. Furthermore, Assaad has per- 
formed quantum Monte Carlo simulations of the SU(A^) 
antiferromagnetic Heisenberg model on the square lat- 
tice and found that the optimal ground state for N = 4 
is a 7r-flux state with gap less spinon excitation and spin 
algebraic correlations [33|]. The lattice translational and 
spin rotational symmetries are preserved in such a 7r-flux 
state. Thus, a system of interacting spin-| fermionic ul- 
tracold atoms at half-filling in a two-dimensional square 
optical lattice is a possible experimental realization to 
observe this interesting algebraic spin liquid state. The 
equal-time spin-quadrupole correlations and the SO (5) 
adjoint operator correlation functions display the same 
power-law behavior due to the SU(4) symmetry 

1 



which in principle can be measured by spatial noise cor- 
relation experiments [34], [35j ■ In the mean field descrip- 
tion, the correlation functions in the 7r-flux state decay 
as l/|r| 4 , because the spinons are effectively described by 
(2+1) dimensional relativistic free Dirac fermions at low 
energy regime. However, the gauge fluctuations are ex- 
pected to enhance the correlations functions as shown in 
the quantum Monte Carlo results (l/|r| 112 ) [33| and the 
calculations including the gauge fluctuations in a large- N 
approximation [201 ]. 



B. Effective model and the phase diagram 

According to the Landau's continuous phase transition 
theory, two phases with incompatible symmetries can not 
be connected by a continuous phase transition (36j |. In 
the previous section, our mean field theory have shown 



that the quantum critical 7r-flux state separates two insu- 
lating long-range ordered states, which are characterized 
by different symmetries, order parameters, and topolog- 
ical characterizations. However, we can not rule out the 
possibility that the quantum critical 7r-flux state actu- 
ally controls a region in the parameter space. Moreover, 
our saddle point approaches to the generalized Hubbard 
model have neglected the particle number fluctuations, 
however, such particle number fluctuations may melt the 
assumed spin ordering at small but finite c 2 and stabilize 
the algebraic spin liquid state. Our system provides such 
a chance to study the quantum critical phase within the 
Landau's paradigm. 

Let us start from the effective Hamiltonian in the 
strong coupling limit [l5[ 

h cS = ,h L ? L T + J 2 E n >* + J 3 E ( L ? 6 ) 2 ' 

{ij),a<b {ij)-, a i.a<b 

(46) 

where J3 oc c<i- At the SU(4) point, J3 = and 
J 1 = J2 = 4i 2 /co, the effective model is reduced to the 
previous SU(4) Heisenberg antiferromagnetic spin model. 
In the spin-quadrupole ordering phase, we should have 
•h ^S> J\ > and J3 > 0, the first term can be neglected, 
and the effective model Hamiltonian becomes an SO (5) 
quantum rotor model. However, in degenerate antifer- 
romagnetic spin-dipole/spin-octupole ordering phase, we 
should have J\ 3> J2 > and J3 < 0, the latter two terms 
can be neglected, and the effective model is reduced to an 
SO(5) Heisenberg antiferromagnetic spin model. Thus, 
this effective model Hamiltonian well describes the ba- 
sic physics of the system from the strong coupling limit. 
In this effective model, the virtue tunnelling processes of 
atoms between lattice sites have been taken into account, 
and the various spin ordering formation can be regarded 
as possible instabilities of algebraic spin liquid state. 



1. Staggered spin-quadrupole ordering state 

Since J 2 > J\ and J 3 > 0, we have to find relevant 
interactions for instability of the 7r-flux state to spin- 
quadrupole fluctuations. Again we use the fermionic 
spinon representations and the effective Hamiltonian can 
be expressed as 

H cS = -J X Y, fUiafipfifi + J - E n >t 
<«?'), a/3 (ij),a 

-•/SEW)'' ( 4? ) 

where J_ = J2 — J\ and spinon number satisfy the con- 
straint J2 Q fi a fi a — 2. Since J_ > and J3 > 0, the first 
term is decoupled according to the valence bond order 
parameters in the 7r-flux state as (|4"!2)) and the latter two 
terms are treated in a mean field approximation. The 
following staggered spin-quadrupole order parameter is 
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introduced 

(-l) i (nf) = A, (-rr«>=0(a^4). (48) 
Then, the mean field Hamiltonian can be obtained as 

H M F = 2JlX E [( COS fc *) fljk-Kk v ,a - ( COS k v) /L/ka 
k , a 

-(2j_ + j 3 )A^/t Q r^/ k _ Q . /3 

k,a/3 

+2J lX 2 N + (2J_ + J 3 )A 2 AT, (49) 

which can be easily diagonalized. The spinon quasipar- 
ticle spectrum is thus derived as 

E± - ±^/4J 2 x 2 (cos 2 k x + cos 2 k y ) + (2J_ + J 3 ) 2 A 2 , 

where an energy gap is induced by the formation of the 
long range spin-quadrupole ordering. By minimizing the 
ground state energy, a set of self-consistent equations de- 
termining the order parameters \ an d A are deduced to 

fi dk x dk y (cos 2 k x + cos 2 k y ) 1 

/"J dk X dky 4(2 J_ + J 3 ) _ 

From these equations, we can easily find the critical con- 
dition, i.e. the boundary between the spin-quadrupole 
ordering and the 7r-flux state. By setting A = 0, we find 
the phase boundary as 

w -3.49^ + 1.49. (52) 

2. Degenerate antiferromagnetic spin-dipole/spin-octupole 
ordering state 

Next we turn to the parameter regime in which spin- 
dipole and spin-octupole fluctuations are dominant. To 
investigate the long-range order formation, the effective 
Hamiltonian can be rewritten as 

H cB = - J 2 Y fiafjafjpfiP ~ J - E L t L f 
(ij),afi {ij),a<b 

+J 3 J2 (Lf) 2 - (53) 

i,a<b 

Since J_ < and J 3 < 0, the valence bond order param- 
eter as the 7r-flux state as (l4"2l can be chosen for the first 
term and a mean field approximation can be made for the 
last two terms. Then the degenerate antiferromagnetic 
spin-dipole/spin-octupole order parameter is introduced 
by 

(Lf) = A, (-I) 1 (Lf) = (otherwise). (54) 
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2- 




-2- 



FIG. 4: The phase diagram for the effective model in the 
strong coupling limit. ASL stands for algebraic spin liquid 
and J- = J 2 — Ji ■ 

Thus, the mean field Hamiltonian is obtained as 

H M F = 2J 2 xX][( COSfc:c )^ka/k- w fe t/ ,a-( COsfc J/)/ka/ka 
k , a 

-(j 3 + 2j_)A^/ kQ ri 5 /3 / k _ Q . /3 

k,a/3 

+2J 2X 2 N- {J 3 + 2J_)A 2 N, (55) 
which yields the spinon quasiparticle spectrum 

Eg = ±yju$ x 2 (cos 2 k x + cos 2 k y ) + (J 3 + 2J_) 2 A 2 , 

(56) 

where again an energy gap is created due to the formation 
of the long-range spin ordering. The corresponding mean 
field self-consistent equations are derived as 

/ 7 dk x dk y cos 2 k x + cos 2 k y 1 
- f (2tt) 2 E+ = 8J2 ! 

n dk x dk y 4( j 3 + 2J_) _ 

Assuming A = 0, the phase boundary between the 7r-flux 
phase and the degenerate antifcrromagnetic spin-dipolc 
/spin-octupole ordering phase can be found as 

^ « ~2 J — - 1.49. (58) 

Now let us map out a ground state phase diagram of 
the model, which is shown in Fig. 4. When c 3> |c 2 | 
and Co 3> t, we have |J_| ~ t 2 |c2 |/c§ <C |c2 1 and J 3 ~ c 2 . 
Thus a stable 7r-flux state with spin algebraic correlations 
may be observed in the region that |c2 1 is comparable to 



t 2 /cq, implying that the competition of the exchange in- 
teraction between different sites and the on-site spin de- 
pendent interaction determines the low energy physics in 
the critical region. Furthermore, two second-order phase 
transitions occur along the two phase boundary curves, 
respectively. We would like to emphasize that there is 
no direct phase transition between the spin-quadrupole 
ordering and spin-dipole/spin-octupole ordering states. 



VI. DISCUSSION 

We have considered Mott insulating phases of spin-| 
fermionic atoms in two-dimensional square optical lat- 
tices at half filling. In addition to the spin-quadrupole 
ordering phase, we found a degenerate antiferromag- 
netic spin-dipole/spin-octupole ordering phase. This is a 
symmetry breaking phase with the Goldstone manifold 
SO(5)/[SU(2)xU(l)]=CP 3 and six Goldstone bosonic 
modes. Compared to spin-quadrupole ordering phase, 
the time reversal symmetry is broken, and there are cou- 
pled transverse bosonic modes, which result in a suppres- 
sion of the density wave velocity in the strong coupling 
limit. The topological properties in the two symmetry 
breaking phases are quite different: the spin-quadrupole 
ordering state contains a non-abelian SU(2) Berry phase 
and is characterized by the second Chern number, while 
the degenerate spin-dipole/spin-octupole ordering state 
contains only abelian U(l) Berry phase and is described 
by the first Chern number. 

Due to these fundamental differences, a continuous 
phase transition from one to another is almost impossible 
(although a first order transition cannot be excluded in 
principle). Between these two phases, the quantum fluc- 
tuations are so strong that an SU(4) 7r-flux spin liquid 
state may be regarded as a quantum critical phase. In 
the low energy excitations of the algebraic liquid phase, 
there emerge U(l) massless gauge field and fractionalized 
spin- 1 gapless spinons. Both the equal-time correlation 
functions of the spin-quadrupole operators and the SO(5) 
adjoint operators display the same power law behavior 
at long distance. We expect these features can be de- 
tected by spatial noise correlation experiments in future. 
We should mention that the above results were obtained 
in the mean field theory and more precise treatment of 
quantum fluctuations may modify the scenario. 
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